Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve perfs for spatial intersection for sql views (refs : #3600) #3888

Merged
merged 4 commits into from
Jan 25, 2024
Merged

Improve perfs for spatial intersection for sql views (refs : #3600) #3888

merged 4 commits into from
Jan 25, 2024

Conversation

bruhnild
Copy link
Contributor

@bruhnild bruhnild commented Jan 16, 2024 •
edited
Loading

refs : #3601 (contributed by @amandine-sahl)

Description

Related Issue

Checklist

  • I have followed the guidelines in our Contributing document
  • My code respects the Definition of done available in the Development section of the documentation
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes
  • I added an entry in the changelog file
  • My commits are all using prefix convention (emoji + tag name) and references associated issues
  • I added a label to the PR corresponding to the perimeter of my contribution
  • The title of my PR mentionned the issue associated

@bruhnild bruhnild self-assigned this Jan 16, 2024
@bruhnild bruhnild requested a review from a team January 16, 2024 09:34
@cypress Cypress
Copy link

cypress bot commented Jan 16, 2024 •
edited
Loading

Passing run #7850 ↗︎

0 24 0 0 Flakiness 0

Details:

Merge 663e4ed into 326b23e...
Project: Geotrek-admin Commit: 05f375ebfc ℹ️
Status: Passed Duration: 02:13 💡
Started: Jan 24, 2024 5:23 PM Ended: Jan 24, 2024 5:25 PM

Review all test suite changes for PR #3888 ↗︎

@codecov Codecov
Copy link

codecov bot commented Jan 16, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (326b23e) 98.35% compared to head (663e4ed) 98.35%.

Additional details and impacted files 
@@           Coverage Diff           @@
##           master    #3888   +/-   ##
=======================================
  Coverage   98.35%   98.35%           
=======================================
  Files         293      293           
  Lines       22174    22174           
=======================================
  Hits        21810    21810           
  Misses        364      364

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@babastienne babastienne changed the title ✨ [Improvements] improve perfs for spatial intersection for sql views (refs : #3600) Improve perfs for spatial intersection for sql views (refs : #3600) Jan 16, 2024
babastienne
babastienne requested changes Jan 19, 2024
View reviewed changes
docs/changelog.rst Outdated Show resolved Hide resolved
@babastienne
Copy link
Member

Otherwise except from changelog that need to be updated, lgtm

@bruhnild bruhnild requested a review from babastienne January 19, 2024 11:55
@submarcos
Copy link
Member

@CodiumAI-Agent /review

@CodiumAI-Agent
Copy link

CodiumAI-Agent commented Jan 19, 2024 •
edited
Loading

PR Analysis

(review updated until commit 16fcf59)

  • 🎯 Main theme: Performance improvement for spatial intersection in SQL views
  • 📝 PR summary: This PR aims to improve the performance of spatial intersection for SQL views. It replaces the previous JOIN operations with LATERAL JOINs to optimize the spatial intersection operations in several SQL views. The changes are applied across multiple files, including those related to trekking, tourism, signage, infrastructure, maintenance, core, outdoor, feedback, and sensitivity.
  • 📌 Type of PR: Enhancement
  • 🧪 Relevant tests added: No
  • ⏱️ Estimated effort to review [1-5]: 4, because the PR involves changes in SQL queries which require a good understanding of SQL performance optimization and spatial intersection operations.
  • 🔒 Security concerns: No security concerns found

PR Feedback

💡 General suggestions: The PR seems to be well-structured and the changes are consistent across all the files. However, it would be beneficial to include some performance metrics to quantify the improvement. This could be in the form of execution times before and after the changes, or some other relevant measure. Additionally, it would be useful to add tests to ensure that the changes do not affect the functionality of the code.

🤖 Code feedback:
relevant filegeotrek/trekking/templates/trekking/sql/post_30_views.sql
suggestion      

Consider adding indexes on the geom columns of zoning_city and zoning_district tables if not already present. This could potentially improve the performance of the ST_Intersects function. [important]

relevant line'+ WHERE st_intersects(a.geom, b_1.geom)'
relevant filegeotrek/tourism/templates/tourism/sql/post_20_views.sql
suggestion      

It's recommended to avoid using functions like array_to_string and array_agg in the SELECT clause of a query as they can slow down the query performance. If possible, try to handle this data manipulation in the application layer. [medium]

relevant line'+ SELECT array_to_string(array_agg(b_1.name ORDER BY b_1.name), ', '::text, '_'::text) AS zoning_city'
relevant filegeotrek/infrastructure/templates/infrastructure/sql/post_10_views.sql
suggestion      

It's good practice to avoid using TRUE in the ON clause of a JOIN operation. Instead, specify the actual condition that should be met for the join. [medium]

relevant line'+ ) f ON TRUE'
relevant filegeotrek/maintenance/templates/maintenance/sql/post_20_views.sql
suggestion      

It's recommended to avoid using functions like ST_Intersects in the WHERE clause of a query as they can slow down the query performance. If possible, try to handle this spatial intersection in a different way. [medium]

relevant line'+ WHERE st_intersects(a.geom_3d, b_1.geom)'

✨ Usage guide:

Overview:
The review tool scans the PR code changes, and generates a PR review. The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on any PR.
When commenting, to edit configurations related to the review tool (pr_reviewer section), use the following template:

/review --pr_reviewer.some_config1=... --pr_reviewer.some_config2=...

With a configuration file, use the following template:

[pr_reviewer]
some_config1=...
some_config2=...
Utilizing extra instructions

The review tool can be configured with extra instructions, which can be used to guide the model to a feedback tailored to the needs of your project.

Be specific, clear, and concise in the instructions. With extra instructions, you are the prompter. Specify the relevant sub-tool, and the relevant aspects of the PR that you want to emphasize.

Examples for extra instructions:

[pr_reviewer] # /review #
extra_instructions="""
In the code feedback section, emphasize the following:
- Does the code logic cover relevant edge cases?
- Is the code logic clear and easy to understand?
- Is the code logic efficient?
...
"""

Use triple quotes to write multi-line instructions. Use bullet points to make the instructions more readable.

How to enable\disable automation
  • When you first install PR-Agent app, the default mode for the review tool is:
pr_commands = ["/review", ...]

meaning the review tool will run automatically on every PR, with the default configuration.
Edit this field to enable/disable the tool, or to change the used configurations

About the 'Code feedback' section

The review tool provides several type of feedbacks, one of them is code suggestions.
If you are interested only in the code suggestions, it is recommended to use the improve feature instead, since it dedicated only to code suggestions, and usually gives better results.
Use the review tool if you want to get a more comprehensive feedback, which includes code suggestions as well.

Auto-labels

The review tool can auto-generate two specific types of labels for a PR:

  • a possible security issue label, that detects possible security issues (enable_review_labels_security flag)
  • a Review effort [1-5]: x label, where x is the estimated effort to review the PR (enable_review_labels_effort flag)
Extra sub-tools

The review tool provides a collection of possible feedbacks about a PR.
It is recommended to review the possible options, and choose the ones relevant for your use case.
Some of the feature that are disabled by default are quite useful, and should be considered for enabling. For example:
require_score_review, require_soc2_review, enable_review_labels_effort, and more.

More PR-Agent commands

To invoke the PR-Agent, add a comment using one of the following commands:

  • /review: Request a review of your Pull Request.
  • /describe: Update the PR title and description based on the contents of the PR.
  • /improve [--extended]: Suggest code improvements. Extended mode provides a higher quality feedback.
  • /ask <QUESTION>: Ask a question about the PR.
  • /update_changelog: Update the changelog based on the PR's contents.
  • /add_docs 💎: Generate docstring for new components introduced in the PR.
  • /generate_labels 💎: Generate labels for the PR based on the PR's contents.
  • /analyze 💎: Automatically analyzes the PR, and presents changes walkthrough for each component.

See the tools guide for more details.
To list the possible configuration parameters, add a /config comment.

See the review usage page for a comprehensive guide on using this tool.

@CodiumAI-Agent
Copy link

Persistent review updated to latest commit 16fcf59

@submarcos
Copy link
Member

@CodiumAI-Agent /improve

@bruhnild
Copy link
Contributor Author

We should keep st_intersects functions as it uses any available spatial index : https://www.cockroachlabs.com/docs/stable/st_intersects. See also : https://mentin.medium.com/which-predicate-cb608b470471

@bruhnild bruhnild merged commit 95bedd3 into master Jan 25, 2024
15 checks passed
@bruhnild bruhnild deleted the mfu-perf-qgis-view-intersection branch January 25, 2024 07:46
@camillemonchicourt camillemonchicourt mentioned this pull request Sep 6, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CodiumAI-Agent CodiumAI-Agent CodiumAI-Agent left review comments

@babastienne babastienne babastienne approved these changes

Assignees

@bruhnild bruhnild

Labels
💫 Improvements
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@bruhnild @babastienne @submarcos @CodiumAI-Agent